The physics-informed neural network approach (PINNs) relies on approximating the solution to a partial differential equation (PDE) using a neural network by solving an associated non-convex and highly nonlinear optimization task. Despite the challenges of such an ansatz, the optimization-based formulation of PINNs provides rich flexibility and holds great promise for unifying various techniques into monolithic computational frameworks. Inspired by Liquid Composite Molding (LQM) process for fiber-reinforced composites and its related multiscale fluid flow structure, we present a novel framework for optimizing PINNs constrained by partial differential equations, with applications to multiscale PDEs. Our hybrid approach approximates the fine-scale PDE using PINNs, producing a PDE residual based objective subject to a coarse-scale PDE model parameterized by the fine-scale solution. Multiscale modeling techniques introduce feedback mechanisms that yield scale-bridging coupling, resulting in a non-standard PDE-constrained optimization problem. From a discrete standpoint, the formulation represents a hybrid numerical solver that integrates both neural networks and finite elements, for which we present a numerical algorithm. We demonstrate that our formulation can significantly improve the convergence properties of PINNs. In addition, we discuss potential applications of the hybrid solver to multi-fidelity learning (by which we understand here the use of physical models and data across two scales) of upscaling operator, which is used for predicting the permeability of fibrous structures in composite materials within LQM.